DOCOHBNT BESOHE 



BO 209 2a7 Ta 810 635 

TITLE Test Ose and Validity: I Bespons^ to Charges in the 

Hader/Nairn Report on ETS. 
INSTITOTIOR Edocational Testing Service, Princeton, H^Jn 
POB DiTE Feb 80 

BOTE 27p«; For a related docoBent, see IM 610 63^4. 

IVIILIBLE FBOH Educational Testing Service, Publications Order 

Services, Oept« E01, Princeton, bj 085^4 1 (free) 

BDBS PBICB HF01/PC02 Plos Postage^ 

DBSCBIPTOBS Adsission Criteria; "^College Adsission; ^College 

Entrance Exasinations; Higher Education; Predictive 
validity; Scores; ♦Test Ose; ^Test Validity 

IDBRTIFIEBS Educational Testing Service; ♦Beign ox £15 (Nairn); 

♦Scholastic Aptitude Test 

ABSTBACT 

The Bairn report. The Beign of EIS, has charged that 
the Bajor college adsissions tests adsinistered by £ducatxonal 
Testing Service (ETS) havr undue influence on ddaissions ^o higher 
education, and that the tests have little value in predicting future 
acadesic perfcrsance. Nairn's claims that the ScJ olastic Aptitude 
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AN OVERVIEW , 

TEST USE AND VALIDITY 

A recent repoit on ETS. written by Allan Nairn and sponsored by 
Ralph Nader, has charged that the major admissions tests 
administered by ETS have undue influence on admissions to higher 
education, and that the tests have little value in predicting future 
academic performance. The Nairn report arrives at its remarkable 
conclusions by misrepresenting the purposes and uses of %e tests, 
and by distorting or ignoring the results of research 

Nairn's claims that tne SAT is a poor predictor of performance in 
college are based on faulty statistics He uses an incorrect value for 
the characteristic validity of the SAT ( 345) since he mistakenly 
averages the separate validities of the two parts of the SAT. rather 
than considering, the validity of the whole test ( 41) After squaring 
that coefficient and doing some further aritnmetic, Nairn comes to 
the conclusion that "for 88% of the applicants an SAT score will 
predict their grade rank no more accurately than a pair of dice" 

Nakn*s interpretation of this statistic is clearly wrong and would 
make no more sense to a layman than to a professional statistician 
Even if the SAT had no predictive validity, it would be no worse than 
a random predictor. However, predictions based on valid information 
will be Ijetter than random predictions, and will certamly be more 
accufO»te more than 5C percent of the time Yet Nairn claims that 
dice wii' be as good a predictor as the SAT for 88 percent of the 
applicants Nairn's dice should be carefully inspected 

Admissions test scoros are used with students* previous grades 
and other information in predicting later academic performance The 
best predictor of college grades is the high school record, but the 
SAT IS nearly as gpod. and the two togelher are btjtter than either 
alone At the graduate and professional school levels, admission test 
scores are nr^ost often the best available predictors of later grades. . 
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Nairn charges that exaggerated and untrue claims are made for ' 
the tests In support of this charge, Natrn quotes incorrectly from 
ETS publications In one instance, he creates a quote, which he 
attributes to ETS, from parts of two separate and conflicting 
statements about aptitude testing from two different sources They 
were used by ETS to illustrate the wide difference of opinion on that 
subject Nairn also quotes from jn SAT publication, deleting words 
and adding others in brackets to transform the meaning of the 
original passage Nairn concludes that testers have assumed a right 
"to define the potential for thinking ' The fact is that ETS claims only 
that tne tests can be a useful measure of knowledge or academic 
ability "No test." state College Board bulletins and m-anuals, 
"predicts with any certainty 'success in life' or is in any way a 
nrieasure of an individual's total worth 

The tests administered by ETS are sponsored and controlled by 
associations of colleges and graduate and professional schools that 
use scores on the tests as one factor in admtssions decision-making 
The tests that are described as aptitude tests were never meant to 
assess innate, unchanging abilities, but to measure learned skills 
They are designed to provide a common basis for evaluation, to 
supplement students' academic records, and to permit students to 
sattsfy admissions testing requirements of institutions across the 
country through taking g single examination at nearby testing 
centers, ^t relatively low cost 

No doubt improvements in admissions can be msrde But Nairn 
does not suggest alternatives that would perform the important 
functions now performed bv admissions tests Because of ihis. and 
because of its misrepresentations, the Nairn report is unlikely to 
contribute to the improvement of admissions to higher education 
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TEST USE AND VALIDITY 

AN INTRODUCTORY NOTE 



The report on ETS commissioned by Ralph Nader and prepared by 
Allan Nairn sets out to discredit the usefulness of tests by attempting 
to prove, among other negative 'findings," that test scores bear little 
relation to success in college or university This conclusion will be 
puzzling to many oeople since it is contrary to the practical 
experience of educators and to the hundreds of research studies 
made all over the country through several decades 

The issue is of importance to educators and especially to the 
students who need to know whether or not the scores r e in fact 
ind'cative of probable college success ETS has always said— as 
have others with experience m this area— that the scotes are by no 
me^ni5 infallible guides and should always be used as only one factor 
in any judgment about whether a student is likely lo do well in college 
or'graduate studies For many years up until the Nairn analysis. ETS 
and others have interpreted the evidence as showing that the 
student's previous giades%re the most 'mportant indicators, but that 
the addition of scores on gobd tests adds significantly to the 
prediction of success 

This paper presents a careful review of the Nairn analysis and 
finds that in fact it is incorrect as experience and research have told 
us for some time, test scores do indeed contnbute useful information 
on which to base admissions decisions The Nairn exercise manages 
to arrive at the opposite conclusions through a combination of • 
selective reporting and statistical error, and presents them with a 
rhetorical flourish that fits the evident intent to demolish the tests 
rather than enlighten the reader 

It IS a matter of public concern that educational decisions be 
based on the most valid available information Clearly, advocacy in 
the areas of testing and admissioti is higly desirable as long as it is 
based on a fair presentation of fact It would be a disservice to 
students, parents, institutions, and the public at large, however, to 
leave unanswered an attack that marshalls half truths to create an 
advocate's case against the use of test scores as one element in 
admissions This paper is intended to help clanfy some of the 
important issues that may have been confused by the Nader/Nairn 
report 

Wilham W Turnbull 
Rresfdent 

Educational Testing Service 




TEST USE AND VALIDITY 

A recen. report on ETS. written by Allan Nai[n and sponsored by 
Ralph Nader, has charged that tlie major admissions tests 
administered by ETS on behalf of associations of colleges and 
graduate^nd professional schbCs-^ constitute a "respectable fraud" 
The report asserts that exaggerated claims have been made for the 
tests.Jhat^s a result the tests have undue influence on admissions 
to higher education, and thcat in fact the tests have little value in 
predicting students' future academic performance According to 
Nairn, most of the' time admissions officers could do as well by 
rolling dice * 

This charge will come as a surprise to the colleges and universities 
that have used scores on these tests as one element in admissions ' 
for decades The faculty and admissions officers at these institut!ons 
know what the test3 are desigji^cJ to do and how they are used 
Moreover, tney have vast experience tp selecting students for 
admission, in teaching these students, and in observing their 
academic performance Apart from this experience, colleges and 
universities have conducted well over one thousario formal research 
studies of the relationship of test scores to the grades later obtained 
by enrolleo students Many of these studies have been conducted m 
collaboration w^th ETS, others by the institutions independently 
These studies show clearly that scores on the tests are useful m 
predicting the grades that students will obtain 

The Nairn report arrives at its remarkable conclusions by 
misrepresenting the purposes and uses of the tests and by distorting 
or ignoring the results of research Despte the fact that accurate 
information on these matters has been widely published and available 
to test users and to the public for a long time, the attention that the 
Nairn report will attract through its exaggerated and sensational 
charges r^uires that the record be set straight 

's 

What the Tests Are Designed to Do 

The major admissions tests administered by ETS are sponsored and 
controlled by associations o» colleges and graduate and professional 
schools that use scores on the tests as one factor in admissions 
decisfon-making Because grades and other information used in 
admissions (such as recommendations) are not directly comparable 
from student to student, these institutions ask that applicants also 
submit test scores, which suppk'ment the other information and 
provide a more uniform basis foi evalution of students* academe 
abilities and achievement The institutions have long recognized the 
advantages to themselves in agreeing to accept results from uniform 
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systems of professionally developed examinations and the 
advantages (o their applicants m having results from a single 
exannination, administered nationwide, acceptable for admissions 
purposes ai multiple institutions In fact, recognition of the benefits to 
applicanis came first, the College Board was founded in 1900 by a 
> group of colleges in response to the problems of secondary schools 
and their students In meeting different and uncoordinated college 
admission testing requirements 

The tests themselves vary in their characteristics The major 
testing program at the college level, the College Board Admissions 
Testing Program, offers a test of scholastic aptitude (the SAT), a test 
of English usage designed primarily for decision-making about course 
placement of entering students, and a series of subject matter 
Achtevenwnt Tests. The Graduate Record Examinations (GRE) 
program similarly offers an aptftude test and achievement tests in 
areas of graduate study Law schools and graduate management 
schools generally use tests that may be broadly characterized as 
aptitude tests— the Law School Admission Test (LSAT) and the 
Graduate Management Admission Test (GMAT) 
' Since the tests that ^re described as antitude tests axe mo*?* 
widely used, most attention focuses on them A common 
misconception is that these tests somehow measure innate, 
unchanging abilities In fact, they measure learned skills They are 
described as aptitude tests because they are not tied to a particular 
course o? study, curriculum or program, and because they p^c 
typically used to assess students' relative abilities tc-^rform well ;n 
future academic work Scholastic aptitudej^^straire most ofie*i made 
up of problems that Jest reading coipprSnension. verbal leasonmg 
and vocabulary matriematt-^i; .easoning. and data interpretation 
They measure intelleciual skills that students are expected to have 
developed throtjgh both school and non-school experiences .part 
from the particular courses of study they may have pursued. These 
skills are also rightfully regarded as broadly applicable to success tr^ 
a variety of future courses of study: since higher education at ail 
ieve^s requires ability to read with understanding, to reason with 
words and numbers, and to calculate and understand quantitative 
relations 

The philosophy underlying much of education in the United States 
today places great stress on providing students with a broad base of 
general education and on delay of academic or professional 
specializf^Hon until a relatively late age. Students throughout high 
school and college are encouraged to acquire a broad acquaintance 
with t^e liberal arts and science Considerable emphasis is placed on 
exploration, flexibility, and choice There is no national syllabus. In 
this context, using tests of developed abilities, rather than relying 
exclusively on subject matter achievemant tests at points of 
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transition from school to college or from college to graduate school, 
IS a consistent, appropriate, and sound policy 

Each of the major admissions tests is made up of multiple-choice 
questions There are principally three reasons for thisrF/rsf, use ci 
these questio; 3 permits a broad Sampling o» problems of different 
kinds in limited testing time Other forms of assessment cannot cover 
as much material in any given time, greatly increasing the risk that 
students will be evaluated in terms of only a few topics that do not 
fairly represent what they know Second, many studies have shown 
that evaluation of essays or other free response exercises is fraught 
with unreliability In practice, each paper can be read by only a few 
individuals, and the grades assigned by different readers to any 
single paper tend to vary a great deal This also poses risks to 
students, since their scores will be affected by the standards and 
.points of view of the particular readers to whom their papers happen 
to be assigned In contrast, the questions on well-developed multiple 
cho'.ce tests are reviewed by many individuals, and there is a 
consensus on the correct answers, scores on tests consisting of a 
number of such questions tend to be much more reliable than scores 
on essay tests Third, multiple-choice testing and the automated 
scoring it permits have a number of practical advantages The low 
costs of the major testing programs to students and timely reporting 
of information are, m large part, a result of this form of testing 

In short, the majot admissions tests are designed to provide a 
common basis for evaluation, supplementary to students' academic 
records w/)/c^ vary in meaning from school to school, a reliable and 
fa^r mean*: of assessing relative student abilities to perform we// in 
future academic v^o^k, one \Nhich is not heavily dependent on -ho 
specific nature of students ' previous academic preparation, and a 
means for students to satisfy admission^ testing requirements of 
institutions across the country thtough taking a single examination at 
nearby testing centers and at relatively /ow cost 

t 

Claims for the Tests 

Nairn's charge of fraud is largely based on a premise that 
exaggerated and untrue claims are made for the tests For example, 
Nairn asserts . 

When ^TS' claims about its aptitude tests are juxtaposed with 
evidence from scholarly Jtudies and internal documents, it 
becomes difficult to conclude that they constitute an even 
minimally prudent construction of the facts (p. 67) 

The ETS consumer is a victim of a false representation of a 
product . This respectable fraud is based on a benignly 
confident adherence to a system of psychometric belief, It is a 
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system characterized by a tolerance for obfuscatton and a 
bedrock assumption about the right and responsibility of the 
mental tester to use multiple-choice questrons to define the 
potential for thinking, and then, regardless of the evidence, to 
draw broad conclusions about the extent to which a person 
possesses It. (p. 58) 

In support of this^charge. Nairn draws on incomplete quotations 
taken from ETS publications. In some cases these quotations are not 
only iakjn out of coniexi and modified, but are incorrectly attributed 
to ETS. For example, the section of Nairn's report from which these 
statements are quoted t)egins with the following sentences: 

Aciiording to an ETS technical manual, '^an aptitude test is . . a 
device for measuring the capacity or potentiality of an individual 
/^toi a particular kind of behavior." Such a device, says ETS. is 
**used to predict success In some occupation or training 
course." (p. 55) . - 

The words in the original text that' have been removed and reduced 
to an ellipsis ( ) are. "commonly thought of as" The context from 
»Miich these staterrients were taken is as follows 

The field of measurement, of course, does not offer any single 
universally accepted definition of an aptitude test. Of several 
which have been offered, perhaps the sharpest contrast is 
afforded by the following According to Cronbach (I960, p 31) 
"An aptitude test is one used to predict success m some 
occupation or training course . In forrrt. these tests are not 
distinctly different from other types . . The test is referred to as 
an achievement test when It is used primarily to examine a 
person's success in past s'uay. and as an aptitude test when it 
IS used to forecast his suco^iss in some future course or 
assignment." However. Ryans and Frederiksen (1951. p.456) find 
that "An aptitude test is commonly thought' of as a device for 
measunng the capacity or potentiality of an individual for a 
particular kind of behavior. In the measurement of aptitude j 
previous experiences or training on the part of the individual is 
assumed either to be lacking or to be constant for all individuals 
comprising tne population considered." 

The Cronbach view, of course, is extremely functional While 
the SAT obviously meets this functional criterion, its 
development over the years since 1926 has also been guided 
largely bv a conception s nilar to the Ryans and Frederiksen 
definition, that "previous experience or training , is 
assumed to be constant for all individuals" (Anooff i97i 
p. 16) \ y . 
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Other statements attributed to ETS and quoted in iHis section of the 
report are contained in the following text by Nairn 



The SAT student bulletin informs teenagers that the test can 
"measure [their] ability to understand what [they] read the 



been showri to be related to college \^rk" Thg LSAT student 
oullelin asserts that the test is a reliable measure "of certain 
mental abilities related to academic performance in law 
schools" In 1977 the GRE stujent bulletin announced that the 
GRE Aptitude Test had been reissued m a new. improve<t model 
which would "measure analytical ability as well as verbal 
and quantitative ability" (p 56) 

For the record, the full description of the SAT excerpted by Nairn was 
as follows 

The SAT is a multiple-choice test made up of separately timed 
verbal and mathematical sections Verbal questions measure 
your ability to understand what you read and the extent of your 
vocabulary Mathematical questions measure your ability to 
solve problems involving arithmetic reasoning, algebra, and 
geometry These abilities have been shown to be related to 
successful academic performance in college The SAT does not 
measure other ktnds of abilities \Nhich may £)e associated with 
success in coUege, such as special taients or motivation 

Your high school record is probably the best evidence of your 
preparation for college Because applicants have taken different 
courses and come from high schools with different grading 
practices, college admissions officers need a common measure 
of ability, such as the SAT However, scores on the SAT are just 
part of the information used m making an admission decision 
(Taking the .SAT, 1978. p 3, emphasis added) 

These are but a few examples of the lack of care and forthnghtness 
by Nairn in the use of quotations from published sources Other 
statements (which Nairn quotes selectively later m his report)!;nrvay be 
found in guidelines published in College Board bulletins and manuals 
for students, high school counselors, and admissions officers For 
example, these guidelines include the following statements 

• test scores, like allotypes of measurements, physical as well as 
psychological, are not perfectly precise and sTiould not be treated 

»as though they were. 

• although admissions test scores are good predictors of 
performance in college, they are not infallible predictors. 

• tests can be a useful measure of knowledge or academic ability^. 



but no test predicts with invkcertainty "success in life" or is in 
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any way a measure of an individual's total worth 

i 

In the same guidelines, admissions offtcers are advised 

Test scores should not be the sole factor in determining the 
admission of an applicant to an institution but should be 
"considered one aspect of the description of an individual in this 
process. Data from the College Board's Admissions Testing 
Pro^^ram is intended to supplement the secondary schocH record 
and other relevant information about the student in assessing 
• competence for college wort; Colleges should view 

admissions tests scores as approximate indicators rather tha^ 
exact measures of a sJudent's abilities and/or achievement 
(Taking the SAl 1978. back cover. ATP Guide for High Schools 
and Colleges, 1379, inside back cover ) 

These statements, even after selection and editorial surgery fay 
Nairn,^clearly indicate thatthe tests m^sure particular; defined" skills 
that have a bearing on future academic p^rfornriance and that 
inferences at|put these skills cannot be made with gnyccertainty How 
Nairn arrived frpm theSe at 9 conclusion that testers have assumed a 
right "to define the potential for thinking.; and then, regardless of the 
evidence, to draw^ broad conclusions about the extent to which a ' • 
person possesses it * is a mystery. 

Nairn questions whether the claims made for the tests "constitute 
an even minimally prudent construction of thefacis" Certain of the 
evidence behind the statements that are made will be discussed 
below. But. it must be noted that Nairn's ciiarge of misrepresentation 
is Itself based on a calculated distortion of the record 

Use of Tests in Admissions 

Consistent with its charges of misrepresentation, the Nairn report 
implies that scores on admissions tests, are given undue weight, and 
that ETS IS somehow an arbiter of admissions^or "gatekeeper" to" 
higher education In fact, colleges and graduate and professional 
schools make their own admissions decisions, each institution using 
Its own criteria. The criteria are decided upon by faculty and 
adminlstrators-at these institutions an3 in the case of many public 
institutions by state boards of education as well The. notion Jhat ETS 
decides admissions is clearly a fallacy 

As indicated above. ETS and the sponsors of the admissions tests 
encourage institutions to weigh va'rious kinds of information in 
admissions decision-making But what are the fatts about teJfl use'^ 
Despite the advice institutions receive, do they generally place an 
undue weight on test scores'? 

At the college level, this does not seem gerierally to be the case 
Most colleges are not selective and admit a large proportion of their 
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applicants A very large majority of students intending to go to 
college do "so "A 1972 study conducted for the federal governnnent 

^^ond that of high school graduates who applied to college, 87 5% 
♦lad been admitted to at least one institution by the end of their 
senior/ yeai (Hilton and Rhett. 1973) A 1978 survey conducted for the 
Amentan Council on Education indicated that 75% of freshmen were 

' attendif^g their first choice college and nearly 95% were attending 

• their first or secoild choice college (Astin. King, and Richardson. ^ 

1978.1) 18) ' 

Even Smong selective colleges, test scores are seldom the most 
important factor t.i admissions, let alone the sole factor A 1979 
su'vey conducted jointly by the College Board and the Ameitcan 
Association of Collegiate Registrars and Admissions Officers- found 
that under 27o of the selective colleges responding to the survey 
indicated that test scores were "the most important factor'' m . 
admissions decision-mak>rig Ninety percent described test scores as 
a "very important factor" or as "one of several factors" {Van Dusen. 
Nelson. Jacobsen, and Ivens. 1979. p 26) These general survey 
- - results^are no cause for complacency- about the issue of proper test 
use But, they do suggest that test scores are much less crucial in 
gdmissic^ns dectstonrs than Nairn supposes 

-Much attention is given in the Nairn report to law schools where 
admissions are generally more competitive than at the college level 
As noted in the Nairn report, the Association of American Law 
Schoots staled m 1973 "for the first timfe in the history^Of United 
States legal. education every accredited law school denied admission 
to applicants who it considered qualified for the study of law" • ' 
l^i^^btedfy, many law schools could fill their classes several times 
overT^ qualified candidates selected from their applicant pools 
This Situation is a function of the number of places in law schools in 
relation to the number of individuals who want to study law. and it 

• would continue to be true if admissions tests were abolished 
tomorrow If the LSAT were removed from the scene, law schools 
would still be selective, but they would have less ijsef ul information 
on which to base selection decsions 

The Nairn report provides iittle evidence that LSAT scores are 
given too much weight in admissions He suggests that a minority of 
law schools may use cutoff scores fen the LSAT. but there is no 
tndtcatior^ whether these cutoff scores (if used) were related to 
minimum admissability requirements or were used to disqualify large 

\ numbers of applicants Nairn discusses at lengtn the fcict that 

predictive indices combining test scores and previoi s grades are 
reported to law schools, but he fails to point out that these reports 
are accompanied by full copies of each student's academic 

, ^ transcript '- 

- ' An overall picture is provided by a table on page 460 of the Nairn 
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report whiCi he adapted from Willingham and Breland (1977) It 
indicates at admission to*ABA accredited law schools is clearly 
related to siudents' undergraduate grades and test scores, but these 
factors are far from decisive Only for students with the lowest or 
with the very highest grades and test scores are the probabilities of 
admission to at least one law school close io zero or one The 
admissions policies of individual ABA-approved law schools are 
described in the Pre/aw Handbook (1979) published by tne 
Association of American Law Schools and the Law School Admission 
Council. These descriptions indicate that the vast majority of law 
schools consider such factors as recommendations, program of 
study, extracurricular activities and community service, work 
experience, personal qualities, accomplishments, and special talents 
as yveil as grades and test scores in making admissions decisions 
However, a few public institutions give little weight to subjective 
assessments, and appear to rely very heavily on previous academic 
grad^ and test scores. 

ETS assists institutions in making appropriate use of test scores 
and other information through regular publications, advisory services 
resdSf^, and other means An example of the last of these is a 
paper prepared by ETS Senior Vice President Wirt^on H Manning 
(1977) for the Caniegie Council on Policy Studies in Higher 
Education Manning proposed an alternate admissions model— one 
that would assist institutions in moderating the weight given to test 
scores. Under this structure. Institutions would make decisions in two 
stages: first. Identifying those students who meet minimum 
admissability requirements in terms of curriculum preparation, 
grades, and test scores; and second, selecting from among the 
admissable students those who in the light of the academic and 
nonacademic evidence, would seem to best advance the educational 
philosophy and objectives of the institution, the profession, and 
society. There is no universe, agreement on this model, though 
variants of it are unJoubtedly in use in many institutions. But. it does 
represent a constructive way to help insure that multiple relevant 
values are consid' ed in admissions. 

What proposals does Nairn advance to help those institutions with 
many nrwre applicants than places to make admissions decisions in a 
fair and wise manner? It is difficult to find in the Nairn report 
constiuctive and practical alternatives to the current use ot tests as * 
one element in admissions. The other kinds of information Nairn 
proposes for use, such as e* ^a^** or personal statements submitted 
as part of the admissions process or reports of special talents and 
accomplishments, are already used by many selective institutions 
However, these institutions also need unif^ m and dependable 
measures of students' developed abilities as an- aid in juoging which 
candidates have the best academlr. preparation and as one means of 




comparing applicants in an equitable way Nairn fails to consider the 
consequences that both irtstitutions and applicants would suffer if 
admissions tests were not used 

In advancing his casfe against the LSAT specifically. Nairn 
overlooks the fact that the best available single predictor of 
academic performance in most law schools is the LSAT— that LSAT 
scores are very generally found to be more closely related fo 
average law school grades than are previous college grades or any 
other known predictor This finding does not suggest that LSAT 
scores should be used alone to decide admissions, for that is 
certainly inadvisable But, ts it not also inadvisable to disregard 
, c\^^.T\y relevant test-based information, and to base Jecisions entirely 
C.I information that research shows to be less predictive of fuiure 
academic performance? There is no more logical basis for weightr.g 
LSAT scores 0% than there is for weighting them 1007© 

Admissions, like most, important decision problems, is a matter ol 
values, judgment, and evidence. The admissions processes of 
selective educational institutions have been worked oi/t through years 
of experience, consideration by responsible educators, debate, and 
research No doubt, changes and improvements m admissions can 
be nr^de. Nairn wants institutions to devalue information provided by 
tests. But. he does not suggest alternatives that would perform the 
important functions now performed by admissions tests Because the 
Nairn report misrepresents the tests and their uses and because it 
offers no serious alternatives, it is unlikely to contribute 
Kjonstructively to improvement of admissions to higher education 

Evidence Supporting Test Use 

Several kinds of evidence supporting use of the teats exist At the 
foundation is an informed judgment by the designers and developers 
of the tests that the tasks sampled by the teats require skills that are 
imp int to competent academic performance — that the tests are in 

f essence academic work samples This judgment is guided by 

experience with particular types of questions or problems in other 
settings and by research showing a relationship between 
performance on these tasks and academic success In addition, this 
judgment is subject to review of independent educators, particularly 
those in the institutions that use the tests. The practice of publishing 
sample tc3t forms with answer key? for each of the major admissions 
testing programs further permits the Widesl possible scrutiny of test 
content. There is considerable agreement among those using the 
tests that the abilities assessed by the reading, verbal reasoning, and 
mathematical problerris contained in the 'tests are relevant to 
successful academic work in their institutions The prevailing view of 

0 the SAT is the one exoressed by William Ambler, dean of admissions 
14 - • 
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at Haverford College (as quoted in Nemweek, February 18, 1980) 
"The iest reflects the words arnj symbols that students must deal 
with In courses every day *^ 

No/ IS this judgment of content>«ttf>^ce-a static one. Each of the 
admissions tests has'changejJ c/er timeln ?ach case, the new 
material has been thoroughly reVjewed by representatives of 
institutions using the tests and ha\been subject to research on the 
ernpirical relationship of performance on the new types of tasks to 
success in academic work (See, eg. French. 1957, French. 1964. 
Flaugher and Rock. 1966, Schrader. 1973, McPeek. Pitcher, and 
Carlson, 1974) 

Many technical studies of the tasts are performed, including 
studies of test difficulty, the consistency of meaning of scores from 
one test form to anolher. the reliability (or stability and precision) of 
scores, and the relationships of scores on the tests to other 
variables. Studies of this last type are often concerned with the 
"validity*' of a test— the extent to which scores relate to othtfr 
measures of educational preparation, development, or achievement 
Important among them are studies of predictive validity, or the extent 
to which test scores are related to future academic performance 
(MSuaity fTwiiJsuf*5o by grades). It is from the results of these studies 
tla\ Nairn attempts to show that use of tho tests is little bettf^r than 
roHtfig dice. 

Prtdictlvt Validity of the admlttlons tests. Though evidence 
from predictive validity studies is by no means the whole story, it is 
important as an indication that test scores bear a meaningful 
relationship to future academic performance Such studies cannot be 
conclusive, however, for ;n many cases predictive studies fail to 
reflect the full value of the tests to selective institutions These 
institutions usejest scores and other academic information to 
identify students who are likely to do well To a large extent, the 
institutions are successful, and the great majority of admitted 
students are fully capable of meeting the requirements of these 
institutions' academic programs; students kicking the basic 
qualifications a\e seldom admitted. Since the prediction studies 
exannlne the relationships of scores to grades for a' restricted, 
capable group of ^nroMsd and persisting students, the observed 
relationships of tests and grsj^es In these studies are lower than 
those that would be 'ounc! for the total group 

Typical results of pre Miction studies based on the test scores and 
grades of enrolled students are shown in Table 1 
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Table 1 



Ctiaractaristic Validity Coef flci«nt$ of Admissions Test Scores aiid 









Median Validity Coefficients 


Admittion 
T«tt 


Typ« of 

School 


No 0< 


Scor«8 


Prtvk>U8 
QPA 


Both 
Pr«d«ctor» 
Combined 


SAT 


Undergraduate 


827 


41 


52 


58 


GRE 


Graduate Arts 
& Sciences 


24-30 


33 


31 




LSAT 


Law 


116 


36 


2b 


45 


GMAT 


Graduate 
Management 


67 


29 


21 


38 



The numbers in the table are ' correlation coefficients" A correlation 
coefficient is an index of relationship generally symbolized Dy the 
letter "r " An r of 00 indicates no relatio-^^hip. (or more precisely no 
"linear" relationship) An r of 1 00 indicates a perfect relationship It 
IS standard professional practice to report predictive validity in terms 
of correlation coefficients 

The numbers shown in th^ table are median values, actual values 
vary from institution to institution Fcr a variety of reasons (mcluding 
the "restriction of range"pro')tem described above), the numbers 
reported in Table 1 are conserve::ve estimates These factots and 
their effects on validity coefficients are discussed later in this paper 
In the discussion immediately followhQ, however, data from the table 
will be used 

The numbers in Table 1 are generally the ones on which Nairn 
bases his claim that rolling dice is jusf about as good as using test 
scores m admissions In fact, he uses a different and incorrect value 
for the characteristic validity of the ^ 345) since he mistakenly 
averages the separate validities of ti o Parts of the SAT. rather 
than considering the validity ol the tOia ^st* Moreover, he decides 
to square the validity coefficients and then 'o multiply the results by 
100 arriving at the numbers 12 for the SAT (bc ^ed on the erroneous 
validity of 345). 1 1 for the GRE. 13 for the LSAl and 8 for the GMAT 
These lie interprets as the percen. cases tor w^'Ch an ETS test 
will provide a prediction more accurate than a ranfjom prediction 
such as by throwing a pair of d'ce Tak* : --^^ r.onriplements of these 
numbers, he states that rolling dice will be as accurate as using test 
scores from 877o to 927o of the time For the SAT specifically. Nairn 




•The charactefistic vandity coefficients foi ihe two parf^ of the SAT ve-bal and mathf rnatfcai are V 
and 32 respectively H does r^ot requi'e advanced stdlislical knowledge to recogm/e that the 
predictive effectiveness of f/ie toiat test wiit be as high o higher than that for either pari aione it this 
w«fe r^ot so one could simply use the part w»lh the higher validity (e g the verbal test with a 
chara'cterist.c v^ilidiiy ol V) and obtain a highe. /aiidity coelfioenl than the average of the coeflicfents 
lor the two p<*ns ( "^iSi 8u« "n tact combining the 'nlo» ."nation from the two parts results m greater 
pfediclive va;»dity {typtcaity 41) ihan does using e ther part separately 
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says that "on the average, for 88 percent of the applicants (though tt 
IS impossible to know which ones) an SAT score will predict thetr 
grade rank no more accurately than a pair of dice ' (p 65) 

It IS Impossible to know how Na»rn arrived at this misconception, 
which was featured prominently both in the report itself and the press 
'elease that accompanied the report The square of a correlation 
coefficient (or r^) »5 a statistic often described as the "percent of 
s»')ared variance" or occasionally as the "coefficient of 
deter mi nat^on" LiMIe can be said of Nairn's interpretation except that 
it IS wrung It is safe to say that no reputable statistics text can be 
found anywhere that suggests that r' indicates the proportion of 
predictions better than chance predictions 

In fact. Nairn's interpretation makes no sense even at an ,ntuitive 
levet If the SAT were invalid, it would be no worse than a random 
predictor, such as a pair of dice If piedtctions based on an invalid 
tost and random predictions were compared for a large group of 
students, the test would give predictions closer to students* obtained 
GPAs for about 50% of the cases, and random predictions would be 
better for the other 50%. Predictions based on valid information will 
be better than random predictions and will certainly more closely 
approximate obtained GPAs for more than 50%, of the applicants 
Yet. according to Nairn, dice will be as good as or better than the 
SAT. a test of proven validity. 88% of the time Like citations of 
evidence in the Nairn report, these dice should be carefully 
inspected. 

In what percent of the cases is tt better to use an admission test 
instead of d'ce'^ A reasonable answer is 100%. for use of test scores 
in e&ch case improves the Itkehhood of making an accurate forecast 
of future performance. Since predictions are inevitably somewhat 
uncertain, sometimes (but well unde' half the time) a blind guess will 
turn out by chance to have been more "accurate" than an estimate 
based on relevant i nform a ti o n . But, thi s i s h a rd l y on-afgufTterm ksr ^ - 
disregarding useful predictive information 

Despite Nairn's misinterpretation of the coefficient of 
determination (r*). it is quite a respectable stati'stic which has a 
definite meaning to statisticians However, there is agreement among 
many who have studied the matter thar the coefficient of 
determination is not an appropriate measure of ihe usefulness of 
tests for selection For example, in a widely read text, Cronbach arK* 
Gleser (1965. p52) state 

Like others who have investigated the selection problem, we find 
no basis for interpreting the usefulness of tests in terms of the 
coefficient of forecasting efficiency or the coefficient of 
determination These indices should not be employed in 
evaluating tests for selection purposes, 
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What can predictive valtdtties tell us about the usefulness of tests for 
selection purposes? A college using the SAT may be particularly 
interested in the average grades that selected students wiiS attain 
Figure 1 illustrates !he average grades that would be expected at a 
college typical of those conducting studies thrcugn the College 
Board's Validity Study Service in 1974 It is a connposite illustration 
based on the typical distributions of test scores and grades found at 
these colleges and on a correlation cf 40, which is just below ine 
characteristic validity of the SAT (In technical teims. it depicts tna 
"linear regression" of grades on test scores ) While there is no way 
to turn a correlation coefficient directly into an easily interpreted 
percentage, Figure 1 indicates that the relationship of SAT scores to 
grades is hardly random-— those students with higher SAT scores are 
typically those who go on to earn higher grades in college As shown 
in the figure, there is also variation in the grades obtained by 
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students with any given score The figure shows grades for the 
middle 50% of the students at each score level Other studems will 
obtain scores outside these ranges While predictions are not certain, 
college faculties and admissions officers find the relationship of tesf* 
scores to grades depicted in Figure 1 agrees with their own 
experience, and they find using SAT scores with grades and other 
information in admissions much preferable to rolling dice 

The usefulness of a test can also be evaluated in "percentage 
correct" terms by determining the proportion of selected or 
unselected students who would perform above a given criterion level 
For example, consider a college with 1000 applicants, but only 900 
places It has to deny admission to 100 students If these students 
were chosen at random, about 50 of the students denied admission 
would have ranked in tne top half of the class fn terms of later 
grades Use of a test with a validity of 41 to select these students 
would reduce the number of incorrect decisions to 21 Use of test 
scores and previous grades with a combined validity of 58 would 
reduce the number to 1 1 These results are based on a standard set 
of statistical tables deve.oped by Taylor and Russell (1939) This 
approach to evaluating validity in terms of the likely number or 
percentage of incorrect decisions is known to Nairn, who refers to it 
in a footnote on pages 419-420 

The tests' contribution to prediction. In considering what tests 
add to piediction. Nairn concludes that "inclusion of SAT scores in 
the prediction process imp'oves the prediction of college grades by 
an average of only five per cent or less" (p 66) For example, in a j 
taole on page 6/^ he shows an "improvement" oi' 5 1 based on data 
from 1974 validity studies Simplifying matters a bit. he states "This 
thin margin is the single thread—the single rational function— on 
which the ETS aptitude empire hangs" (p 66) 
^ In arriving at a figure of five percent, which in his chaprer title he 
calls "five percent of nothing." Nairn switches statistics— from r' to 
another index In fact, he uses the other statistic referred to by 
Cronbach and Gleser, above. th e--'coe fficient of forecasting 
efficiency." which is equal to l-Vl-r^ The actual coefficients for the 
1974 data (not reported by Nairn) are 13 4 for high school grades 
alone and 18 5 for tests and grades combined, which do indeed differ 
by 5.1 This index has a theoretical range from 0 (for random 
prediction) to 100 (for perfect prediction) and can be interpreted in 
percentage termsf though the interpretation is not a particularly 
simple one Since "perfect prediction" cannot be attained,. it is 
reasonable to consider the contribution of tests in relation to the;.level 
of prediction offered by grades alone Simple anthmetic (5 1 divided 
by 13 4) indicates that combining test scores with grades increases 
the value of Nairn's chosen index by 36%, not 5% 

Nairn does not attempt a similar demonstration for other tests. 





such'as the LSAT. because he says (without presenting any data) that 
"the relative predictive ability of grades and scores swings back and 
forth from one time period to another" (p 66) If he had simply taken 
the characteristic law school validities and had computed fiis index 
values from them, he would have *ound coefficients of 3 2 for 
previous grades alone and 10 7 for grades and LSAT scores 
combined— *a difference of 7 5 and a oercentage improvement of 
234% It IS hard to understand why Nairn does not do this, because 
he uses tne LSAT validities from Table 1 of this paper {in modified 
form) in other sections cf nis report 

In Nairn's original terms of reference (r^) it is also difficult to arrive 
at a 5% improvement figure Using the correlations from Table 1 of 
this paper, the results are acproximately as follows for the four 
admissions tests 





pr*vlotJSGP'4 


ftortott 
and QPA 

cofflMrMd 


inr* 


improv«fn«ftt 


SAT 


52'= 27 


58^ = 34 


07 


07/ 27 = 26% 




31'= 10 


45^= 20 


10' 


10/ ^Q = 100% 


LSAT 


25'= 06 


45^- 20 


14 


14/Qg= 23o% 


GMAT 


21'= 04 


38-^ 14 


10 


10/q^= 250% 



it snould be clearly understood that ETS makes no claim that the 
LSAT improves prediction by 234% or the GMAT by 250%, though 
interpretations in terms of Nairn's varying statistical indices lead to 
these conclusions These results, in fact, are not helpful in 
understanding the relative contribution of grades and tests to 
prediction 

It IS important to recognize that test sco.es and previous academic 
records each contribute uniquely to prediction One meaningful way 
to assess the contribution of several variables to prediction is to 
examine the weights that must be applied to each variable to obtain 
optimum overall prediction Consider a college at which SAT scores 
alone have their typical validity of 41 and where high school grades 
alone have their typical validity of 52 In usmg^ these predictors 
together to obtain a typical overall. predictive validity of 58, the test 
scores would be given 39% of the weight and grades would be given 
61% of the weight At the graduate and professional school level, the 
validity study results would lead one to place even more weight on 
tests than on grades m order to obtain optimum prediction But, this 
IS not a decision that should be based solely on the results of validity 
studies. 



Prior grades reflect the pooled judgment of a number of instructors 
who have evaluated students' work on tests, term papers, class 
projects, recitalions. and other aspects of academia'work 
Admissions tests reflect student performance on a carefully selected 
sample of intellectual tasks, scored in a highly consistent way across 
fndividuals. Students' prioi academic records are properly given great 
weight in admissions. Likewise, scores on the tests are property 
weighted in these decisions Although tests measure a narrower 
range of abilities than grades, they are also valid. Because test 
scores have a consistent meaning for all students, they help to 
redress the unfairness that would occur if grades from a variety of 
sending schools with different grading standards were used alone It 
is very largely this fairness, as well as a desire to select students 
who will perform well academically, that motivates institutions to use 
the tests. 

What is'obscured in the Nairn analysis is that deciding on 
admissions criteria is fundamentally a matter of judgment. These 
judgments take into account the relevance, dependability, 
comparability across individuals, comprehensiveness, and validity of 
the information considered for use. It is not a matter *^3t should be 
decided in simple, absolute terms relying entirely on the results o^ 
limited statistical studies If the decision were made in these terms, 
test scor<^s would probably be given greater weight than is typically 
the case currently, for the studies show them to be nearly as valid as 
or, in nnany cases, nnore valid than grades Fortunately, admissions 
officers and admissions committees generally take a restrained view 
of validity study results and use considerable judgment and 
discretion in weigMhg other kinds of evidence. 

Probltms in interpratation. As noted earlier, the estimates of 
predictive validity provided by the i/alidity studies are conservative 
estimates. This is due. in part, to the effects of selection and the 
resulting restriction in range of abilities for selected groups. 
Formulas for adjustrng for restriction of range are available, but their 
use is based on untested assumptions Practical examples show, 
however, that the effects of restriction in range can be severe. A 
dramatic example can be found in a report by Thorridike (1947. p. 66) 
on World War II selection of Army Air Force pilots. The entire group' 
of would-be pilots took the test, and the entire group was allowed fo 
proceed tnrough training before a final selection was made Validity 
of the test was 18 for the selected group; for the unselected group 
as a whole, the validity was .64. 

An exannple closer to the college admissions scene can be fopnd 
in a study by Jackson (1977) which reported the correlations of 
scores with high school grades. For groups of students selected 
and enrolled in colleges participating in College Board validity 
studies, the typical correlations of vhigh school grades with SAT verbal 
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ancHnathematlcal scores were found to be 29 and 31, respectively 
For general samples of SAT test takers, the typic-il correlations 
between self-reported high school grades and SAr verbal and 
mathematical scores were 50 and 53. respectively (Note that these 
correlations are for separate scores on the two parts of the SAT The 
typical correlation of combined SAT scores with hgh school grades 
for unselected groups of high school juniors and seniors is .56.) 

Another factor that attenuates the validity coefficients is "criterion 
unreliabHity"— the fact that the grades which are predicted are 
imperfect measures of academic performance. Students take 
different courses and programs of study Grading standards vary 
from program to program and from instructor to instructor, and even 
the grade a student receives from a particular instructor in a 
particular course is generally a matter of judgment The grade point 
averages that students attain are used as criterion measures in 
validity studies because they useful and reasonable outcome 
measures and because they are meaningful to faculty members, not 
because they are viewed as ideal or perfectly consistent measures of 
academic performance in college. 

Again, formulas are available for adjusting validity coefficients for 
criterion unreliability, but they are seldom used because the degree 
of criterion unreliability is not precisely known However, empirical 
studies show that the effects on validities may be substantial At the 
University of California, Goldman and Slaughter (1976) estimated 
what the combined validity of hiqh school grades and SAT scores 
would be if QPAs for all students were based only on grades earned 
in the same college courses^ They found a marked increase in 
validity (from about 44 to 70) Because of technical difficulties in 
this study and because of uncertainty about the degree to which 
results can be generalized, the results may overstate the typical 
effects of "criterion unreliability * on predictive validity Nevertheless, 
these results and those of previous research by the senior author led 
these researchers to conclude "In sum we believe that the validity 
problem in GRA prediction is the result of the GPA criterion, rather 
than the tests used as predictors." 

The two factors discussed here — restriction of range and criterion 
unreliability — probably explain, in large part, why performance at the 
graduate and professional level appears to be less predictable than 
performance at the college level The range of talent at the graduate 
level is more restricted than at the college level, ^nd grades are 
generally concentrated into only a few categories (e.g.. A, B and C). 
instead of the fl/e categories that are more often used at the 
undergraduate level 

Nairn seems to be well aware of these problems in interpretatior^ 
of validity study results and of the standard approach to research on 
test validity, for there is a discussion of these matters in a footnote 
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oh pages 416-420 Unaccountably, he dismisses these concerns 
without stating clearly why they are »jnworthy of consideration. He is 
correct, however, in stating that testing agencies avoid adjusting 
compute validity coefficients, since it is judged preferable to report 
the more cwservative estinnates wf-.ich are understood as such by 
professional users 

Standartfe for valMtty. What are reasonable standards for test 
validities and to what extent do the major admissions tests meet 
these standards? Because ideal validity studies cannot be performed, 
the valic*ities are not precisely known Even without taking into 
.account the various factors that attenuate validity coefficients, 
however, the obtained results indicate that the t€|sts are useful , 
predictors. Figure 1 i(lustr4tes that a test with a vaiidlty of .40 is quite 
useful in identifying sti:dents who are likely to do well in academic 
course work at the college level. 

The predictive validities of tests can also be judged in relation to 
those of the usual alternative predict( rs. At the college level, the 
characteristic validity of the SAT ( 41) is not a great deal lower than 
that of high school grades (.52). In fact, these values vary from 
college to college, and In about 25% of the colleges the SAT is a 
better predictor than high school grades. At the graduate and 
^^fesslonal levels, as noted, the tests most often have somewhat 
, higher vatidlties ttian does urWergraduate GPA On the whole, the 
tests are atx>ut as useful m predicting future academic performance 
as previous academic grades, the predictor that is most widely used 
and accepted 

"Nairn prefers to judge the test validities in relation to "perfect 
prediction** which is. of course, unattainable Human behavior and 
performance are notoriously variable, many factors (academic and 
perso^jal) affect students* achievement, and grades are an Imperfect 
measure of academic outcomes. Anything approaching perfect 
prediction, in fact, would require massive Intrusion Into people's lives 
to obtain the needed predictive data and stnct regimentation of later 
academic experiences and other aspects of life Colleges do not seek 
certainty in their admissions programs Instead, they atternpt to make 
the best judgments possible using the most relevant available prior 
information Nairn believes this is not good enough, but he does not 
indtcatewhat educational institutions, faced with a need to make 
intelligent decisions, should do. 

It mayjae recalled that Nairn expressed doubt about whether the 
claims made for the tests are supportable. If It were claimed that the 
tests are perfect or if it were recommended that test scores be used 
• alone to decide admissions, there would be grounds for grave 
concern. But. the claims made in test program publications are that 
the testf measure particular skills in the areas of readiqg. 
vocabulary, verbal reasoning, and mathematics that have been 
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shown to be related to future academtc performance, it is 
recommended that test scores be used to supplement students' prior 
academic records In view cf the judgment by educators that the 
tests do measure important academic skills and in view of abundant 
research which clearly shows a useful predictive relationship of test 
scores to later grades, it is very difficult to conclude that Nairn's 
charges of fraud and misrepresentation themselves "constitute an 
even minimally prudent construction of the facts" 

Beyond Academic Selection 

Though Nairn gives much attention to the issue of test validity, he 
seems to recognize that his demonstration is unconvincing, for he 
also criticizes college grades, the criterion predicted by the SAT 
(pages 80-82) In his catch phrase— "5% of nothing*— the word 
'"nothing" rej^resents college GPA It is not a mattei of imperfect 
measurement but what GPA stands for Here again there s^ems to 
be a difference between Nairn's values and those of the colleges and 
unive^rslties, a difference far more fundamental than any dispute over 
~predjctive validity Colleges clearly value academic talent and 
excellence as qualities to be sought ana nurtured Nairn's values are 
not precisely clear, but his report gives little attention to the 
importance of learning and academic achievement This may help to 
explain why Nairn's views on how to go about admissions are so 
divergent from those of college and university faculties 

These comments are perhaps somewhat unfair to Nairn for 
probably he does share many of the values that his mentor, Ralph 
Nader, expresses in a preface to the report There Nader indicates 
that he would prefer to give more attention to characteristics not 
measured by multiple-chcxe tests, such as "judgment, wisdom, 
experience, creativity, idealism, determination, or stamina" (p xn) 
Elsewhere m the report. Nairn suggests that more weight should be 
given to non-academic activities and accomplishments 

Though few in the colleges would agree with abandonment of 
current academic measures, there is unquestionably much sentiment 
that other personal qualitios like those cited by Nader should be 
given more systematic attention in admissions Though these 
qualities sre highly valued and sought, they are difficult l«\assess m 
any but me most subjective terms 

ETS and the client groups it serves have for many years funded, 
conducted, and published research dpoigned to help colleges make 
better use or information on personal qualities in admissions (See. 
- - . . for e;(ample. Anastasi, N^eade. and Schneiders. 1960, Davis. 1964, 
and Baird. 1979. several of which are cited m the Nairn report.) 
Currently the College Board and ETS are jointly engaged m a major 
research project in collaboration with nine colleges to develop 
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Improved assessments of personal qualities, special talents, skills, 
and S'i^nificant accofDplishments As part of this project, the 
relationships of these measures to student development over the four 
college years^development broadly conceived in both r^cademic and 
non-academic terms— will be studied 

tTS has no quarrel with Nader and Nairn's advancing their views 
concerning appropriate criteria for use in selective admiss.ons But it 
cannot be agreed that Nader and Nairn's values are the only values 
Of that tnese values should be substituted for the considered 
judgment of tne faculties and administrators of the educational 
institutions concerning the proper weighing of academic and non- 
academic information in admissions Nor are the tactics employed by 
Nader and Nairn in advancing their point of view designed to lead to 
thoughtful debate of these issues. 

Institutions using admissions tests believe that these tests have 
important benefits both to themselves and to their applicants They 
do not view this as a narrow technical issue, \homh in large par; 
statistical research confirms that the tests are usiful They are also 
well aware that the information and assessments of student 
characteristics they use in admissions are not infallible Opportunities 
to contribute constructively to improvement of testing, assessment 
and admissions are great The Nairn report, which misrepresents 
many important facts and which offers no serious alternatives 
^ contributes very little to such improvement 
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